A double-blind experiment on interannotator agreement: the case of dependency syntax and Finnish

Manually performed treebanking is an expensive effort compared with automatic annotation. In return, manual treebanking is generally believed to provide higherquality/value syntactic annotation than automatic methods. Unfortunately, there is little or no empirical evidence for or against this belief, though arguments have been voiced for the high degree of subjectivity in other levels of linguistic analysis (e.g. morphological annotation). We report a double-blind annotation experiment at the level of dependency syntax, using a small Finnish corpus as the analysis data. The results suggest that an interannotator agreement can be reached as a result of reviews and negotiations that is much higher than the corresponding labelled attachment scores (LAS) reported for stateof-the-art dependency parsers.